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1_. _ n  'REPORT  SUMMARY. 

The  purpose  of  the  Image  Transformation  Study  is  to  develop 
techniques  for  converting  oblique  aerial  photography  to  the  ground- 
level  viewing  perspectives  that  would  be  seen  by  an  observer  as  he 
moves  around  within  the  imaged  area.  (  This  report  documents  the  first 
six  months  of  the  investigation. 

The  approach  to  the  problem  included  five  areas: 

•  A  literature  search  was  performed  to  identify  previous 
investigators'  progress. 

•  Appropriate  aerial  imagery  was  acquired  and  digitized. 

•  The  mathematical  relationships  defining  the  required 
image  transformations  were  derived. 

•  Computer  programs  were  developed  to  perform  the  mathe¬ 
matical  manipulations. 

•  Representative  ground-level  perspective  views  were 
created  using  the  computer  programs. 

Aerial  photographs  were  taken  over  three  areas  of  Santa 
Clara  County.  Photographs  of  each  area  were  taken  from  four  oblique 
directions  as  well  as  overhead,  and  were  taken  in  both  morning  and 
afternoon  light  to  ensure  optimum  viewing  conditions.  The  12-inch 
focal  length  camera  was  at  3000  to  3500  foot  altitudes  and  used 
9-inch  wide  black  and  white  film.  Portions  of  the  film  have  been 
digitized  (i.e.,  converted  to  the  format  of  a  digital  image  processing 
computer  system)  at  a  resolution  corresponding  to  a  3-inch  ground 
picture  element  size.  The  digitized  imagery  has  been  used  in  subse¬ 
quent  stages  of  the  study.  Section  6.1  has  examples  of  the  aerial 
photography. 
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1.  —  Continued. 


A  mathematical  technique  for  deriving  a  camera  model  (i.e., 
the  camera's  position  and  orientation)  from  a  photograph  taken  by  the 
camera  in  conjunction  with  ground  truth  data  was  developed  and  applied 
in  order  to  obtain  this  information  used  in  the  image  transformations. 
Section  3  presents  the  technique. 

The  geometric  projections  used  in  this  study  are  perspective 
transformations.  The  one-camera  perspective  transformations,  which 
relate  image  and  spatial  information,  were  derived.  The  two-camera 
perspective  transformations,  which  relate  imagery  viewed  from  one 
camera  position  to  that  which  would  be  viewed  from  a  second  camera 
position,  were  also  derived.  The  perspective  transformations  are 
presented  in  Section  2. 

A  computer  program  was  written  to  interactively  extract  and 
manipulate  a  stick  figure  model  that  represents  the  underlying  three- 
dimensional  structure  of  a  photographed  scene.  The  three  dimensional 
stick  figure  model  contains  the  information  needed  to  convert  digital 
imagery  from  one  perspective  view  to  another.  The  stick  figure  model 
program  is  described  in  Section  4. 

A  second  computer  program  was  written  to  perform  the 
perspective  transformations  of  the  gray  level  or  intensity  information 
in  a  digital  image  to  convert  it  from  the  overhead  oblique  viewing 
position  to  ground  level  horizontal  viewing  positions.  This  program 
is  discussed  in  Section  5. 

The  computer  programs  were  used  on  the  digitized  images  to 
produce  several  examples  of  output  images.  These  images  are  realistic 
in  their  rendition  of  the  scenes  and  maintain  the  same  resolution  as 
the  original  imagery.  The  examples  are  presented  in  Section  6. 
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2_. _ PERSPECTIVE  EQUATIONS. 

The  perspective  equations  are  the  mathematical  description 
of  the  process  of  imaging  with  a  system  of  ideal  cameras.  An  ideal 
camera  has  a  flat  focal  surface,  called  the  imaging  plane.  A  point  in 
space  (the  object  point)  is  imaged  onto  this  plane  at  the  intersection 
of  the  line  from  the  lens  point  to  the  object  point  with  the  imaging 
plane  (the  image  point) . 

The  one-camera  perspective  equations  (Section  2.1)  relate 
the  object  point  to  the  image  point.  The  two-camera  perspective 
equations  (Section  2.2)  relate  the  image  point  in  one  ideal  camera 
to  the  image  point  in  a  second  ideal  camera. 

2 . 1  _ One  Camera  Perspective  Equations. 

2.1.1  _ Definitions. 

The  perspective  equations  describe  the  photographic  process; 
namely,  the  conversion  of  three  dimensional  data  into  a  two  dimensional 
representation.  Figure  2-1  shows  a  point  (x,y,z)  in  the  spatial  or 
global  coordinate  system  and  the  corresponding  picture  coordinates 
(x  , z  )  as  imaged  by  the  camera.  Note  that  the  picture  coordinate 

r  r 

system  is  centered  on  the  optical  axis  a  distance  equal  to  the  focal 
length  from  the  lens  of  the  camera.  The  camera  is  positioned  at 
spatial  coordinates  (xo,yo,zQ)  and  has  an  orientation  given  by  its  pan, 
tilt  and  roll  angles  (0,<J>,t).  These  angles  are  defined  as  follows. 
Consider  a  coordinate  system  with  the  camera  at  the  origin  and  point¬ 
ing  along  the  Y  axis.  Pan  is  a  rotation  of  the  coordinate  system 
about  its  z  axis.  Tilt  is  a  rotation  of  the  coordinate  system  about 
its  X  axis.  Roll  is  the  rotation  of  the  coordinate  system  about  its 
Y  axis.  The  rotations  are  counter  clockwise  and  must  be  done  in  this 
order.  The  overall  transformation  of  the  camera  from  its  initial 
orientation  to  its  present  orientation  can  be  expressed  as  a  rotation 
matrix,  namely 
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2.1.1 


--  Continued. 


R  = 


ee$  = 


COS  l 
0 

L-sinc, 


0 

1 
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-cost})  sin6 


sin  e" 

'1 

0 

0  ' 

COS0 

sine  0 

0 

0 

cost}) 

sint}> 

-sinfi 

cost-  0 

cose  - 

.0 

-sin4> 

cosp. 

.  0 

0  1 

cose 

sin0  -  sine 

sin<}> 

cos6 

sine  cost 

cost}) 

COS0 

sinC 

3  -sine 

sin0  -  cose 

sintj 

cosB 

cose  cos ' . 

:  2-1 ) 


,-i 


D-i 

R  represent 


R  is  a  unitary  matrix,  i.e.,  R  =  R  .  The  columns  of 
the  unit  vectors  in  the  camera  coordinate  system  (the  picture  coordi 
nate  system)  relative  to  the  spatial  coordinate  system: 


cose,  cose  +  sine;  sin<}>  sin8‘ 
cose,  sin0  -  sine;  sin<}>  cos8 
Lsine  cos<{) 


(2-2) 


-cost})  sin0 
cost})  COS0 
sin<}) 


(2-31 


-sine  cos0  +  cose  sint}>  sin8 
-sine  sin8  -  cose  sin<j>  cos8 
L  cose  cost}) 


(2-4) 


2.1.2 


Forward  Transformation. 


Any  spatial  point  (x,y,z)  can  be  described  by  a  vector  from 
the  spatial  origin  to  the  point  in  question.  Thus 
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2.1.2 


--  Continued. 


v  =  y 


(2-5) 


and  the  camera  position  can  be  described  by 


vo  =  P'0 


(2-6) 


If  the  point  lies  in  the  image  plane,  then  it  can  be 
expressed  as 


%  ‘  %  *  Vl  +  f®2  +  V3 


'2-1) 


where  and  z^  are  the  coordinates  of  the  point  in  the  image  plane. 
This  equation  simply  says  that  the  vector  v^  can  be  expressed  as  the 
vector  sum  of  vq  plus  the  offset  of  the  point  from  the  camera  position. 

Now  consider  the  triangles  formed  by  the  optical  axis,  the 
ray  from  the  camera  through  point  <x  ,o,z  )  to  point  (x,y,z)  and  the 

r  F 

perpendiculars  dropped  from  these  points  to  the  optical  axis,  (see 
Figure  2-1).  These  two  triangles  are  similar.  Thus 


v_  -  v. 


projection  of  (v-v  )  perpendicular  to  the  optical  axis) 
iprojection  of  (v-v  )  along  the  optical  axis 


(2-8) 


ESL-PR  3  39 


i 

2.1.2  —  Continued. 


In  a  similar  fashion,  the  components  of  the  vectors  (v  -v  )  and  (v-v  ) 

poo 

projected  relative  to  the  picture  coordinate  system  also  form  similar 
triangles.  Thus 


x  = 
P 


f  <fe1>  v-vo> 

v-v  > 
2  o 


(2-9) 


and 


z  = 
P 


f  <S3,  v— vq> 
<^2 ' 


(2-10) 


where  the  bracket  notation  represents  the  inner  product.  Equations 
(2-9)  and  (2-10)  are  the  single-camera  perspective  equations  that 
describe  the  transformation  from  the  spatial  coordinate  system  to  the 
picture  coordinate  system. 


These  equations  can  also  be  recast  into  a  different  repre¬ 
sentation.  Consider  the  matrix  N  formed  by  concatenating  the  trans¬ 
posed  unit  vectors,  namely 


'  cos^  cos©  +  sin£  sin<f>  sin0 
-sin£  cos0  +  cos£  sin$  sin© 
-cos<t>  sin0 


cos£  sin0  -  sin^  sin<$>  cos0 
-sin£  sin©  -  cosS  sin<j>  cos0 

COSlJ)  COS0 


sini;  cos4> 
cos£  COS(() 
sin<j> 
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2.1.2  --  Continued. 


11 

N12 

N13 

21 

N2  2 

N23 

31 

N32 

N33 

(2-11) 


Equations  (2-9)  and  (2-10)  can  now  be  expressed  as 


f  I  Nj j  <v-vc) 3 


r  N2j  (V-V^J 


Z  = 

P 


f[Nn(x~xo)  +  Ni2(yy0)  +  "i3{z-zo[ 
-N  21U-xor+  N22(y-yo)  +  N23(z-zo) 


f  *  N3j  (v-Vo)j 
^  N2j  o)j 


(2-12) 


f  | 

N31(x“xo) 

+  N32(y'yo) 

+  N33(z-zo)] 

.N21(x*xo} 

+  N22(y-y0) 

*  N23(z-zo>] 

(2-13) 


2.1.3 


Inverse  Transformation. 


Given  a  point  (x  , z  )  in  the  picture  coordinate  system,  it  is 

Jb 

now  desired  to  find  the  corresponding  point  v  in  spatial  coordinates. 
Equations  (2-9)  and  (2-10)  can  be  expressed  as 


<(xp&2  -  f^)  ,  (v-vq)  >  =  0 


(2-14) 


m*T  4 
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2.1.3  —  Continued. 


<(z  j2  "  f^3 )  '  (v-vq)  >  =  0  (2- 

From  these  equations  it  is  apparent  that  the  vector  (v-v  )  is 
orthogonal  to  both  ( 2  ”  and  (z  ^  ~  ^3)  '  anc*  is  therefore 

proportional  to  their  cross  product: 


(2-15) 


V-VQ  =  y(xp^2  -  f§1)  X  (zp&2  -  f§3)  , 

where  y  is  an  arbitrary  constant  of  proportionality.  Using  the 
orthonormality  property  of  the  unit  vectors,  one  then  obtains 


(2-16) 


(2-17) 


which  is  the  inverse  transformation  required.  The  free  parameter 
X  =  -yf  represents  a  distance  function  along  the  optical  axis  and 
occurs  because  a  2-P  vector  is  being  projected  into  3-D  space. 


Equation  (2-17)  can  be  expressed  in  a  different  form  for  use 
in  subsequent  sections  of  this  report.  Define 


The  tilde  is  used  to  distinguish  this  vector  from  the  spatial  vector 
described  by  equation  (2-7) .  It  represents  the  picture  coordinates 
of  the  point  relative  to  the  camera.  If  we  now  define  a  matrix 
-1  T 

M  =  N  =  N  ,  i.e. 
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--  Continued. 


\l 

M12 

M13 

’Nll 

N21 

N31 

pij- 

M21 

M22 

M2  3 

= 

N12 

N22 

N32 

_M31 

M32 

M33. 

,N13 

N23 

N33. 

(2-18) 


Then  equation  (2-17)  can  be  expressed  as 


or 


v  =  v  +  X  M  \> 
o  p 


(2-19) 


x  =  x  +  AIM  j(v  ) 

3  J  *3 

=  XQ  +  UMnxp  +  M12zp  +  M13f) 


(2-20) 


y  =  yo  +  X£M2 . (vp> 

=  yQ  +  X(M21xp  +  «22Zp  +  M23fl 


(2-21) 


Z  =  zo  +  XIM3j(vp) 


=  ZQ  +  A(M31xp  +  M32zp  +  M33f) 


(2-22) 


2.2 


Two-Camera  Perspective  Equations. 


The  central  concept  explored  in  the  study  is  the  transforma 
tion  of  imagery  taken  from  one  position  to  what  would  be  seen  from  a 
second  position.  The  mathematical  relationships  expressing  this 
transformation  are  the  two  camera  perspective  equations. 
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2 . 2 _ --  Continued. 

Given  a  camera  at  position  v  ,  with  a  focal  length  f, 

forward  projection  matrix  N  ,  and  inverse  projection  matrix  M  ,  and 

a  second  camera  at  position  v^,  focal  length  f',  and  projection 

matrices  H'  and  it  is  desired  to  derive  an  expression  for  the 

position  Vp  in  the  second  camera's  focal  plane  corresponding  to  an 

object  imaged  at  vp  in  the  first  camera's  focal  plane.  It  is 

impossible  to  solve  this  problem  without  making  assumptions  about 

the  scene  being  imaged,  because  there  are  infinitely  many  points  in 

space  which  may  be  imaged  at  a  particular  point  v  ,  generating  an 

infinite  locus  of  points  v'  .  Figure  2-2  (a)  shows  that  v  corre- 

P  P 

sponds  to  a  ray  in  space;  the  ray  generates  a  line  in  the  second 

camera's  focal  plane.  If  we  assume  that  the  scene  contains  a  known 

surface  shape,  then  the  problem  becomes  solvable,  as  a  unique  spatial 

point  is  imaged  at  v  ;  if  there  are  no  obstructions,  the  same  point 

r 

is  imaged  at  vp.  The  situation  where  the  imaged  scene  is  a  plane  is 
depicted  in  Figure  2-2  (b) ;  it  can  be  seen  that  the  problem  is  now 
solvable.  Making  the  planar  assumption,  the  following  derivation 
applies. 


Equation  (2-19)  states  that  any  point  v  on  the  ray  imaged 
at  vp  can  be  represented  by 

v  =  VQ  +  XMvp  (2-19) 

The  plane  can  be  represented  by 

Ax  +  By  +  Cz  +  D  =  0  (2-23) 

where  A,  B,  C,  and  D  are  real  numbers,  or  in  vector  notation  as 

<a,v>  +  D  =  0  (2-24) 
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a.  The  ray  defined  by  vp  generates  a  line  in  the 
second-camera  focal  plane. 


/ 

/ 


b.  Imaging  a  plane,  v  uniquely  determines  v' 

P  P 

Figure  2-2.  Two-Camera  Perspective  Imaging 


I 
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2.2 


—  Continued. 


T  'j1 

where  cf  =  (A,B,C,)  ,  and  v  =  (x,y,z)  is  the  point  at  which  tht 

imaging  ray  intersects  the  plane. 


Substituting  (2-19)  in  (2-24)  and  solving, 


<a,v  >+D 
o 

<a , M  Vp> 


Substituting  (2-25)  into  (2-19) , 


(2-25) 


v  =  v 


<a,v  >+D 
_ o 

<a,  M  v  > 
P 


M  v 


(2-26  ) 


The  second  camera  images  v  at  a  point  in  its  focal  plane: 


<a,v  >+D 

v'  =  v - 2 -  M  'v' 

^  °  <a,  M  'v' >  P 


(2-27) 


Equating  (2-26)  and  (2-27)  and  solving, 


VP 


<a, M  v^>  M 

<a, v  >+D 
o 


.-1 


<vo-V>  - 


<a,v  >+D 

- 2 -  M  v 

<a ,  M  v  > 


(2-28) 


Since  the  third  coordinate  of  v'  is  f',  equation  (2-28)  implies  that 

P 


<a,  ■  v  >  ^  , 

f  '  =  — - - -2—  ^  ,  v  - v '  — 

<a,v'>+D  3  °  ° 


<a , vq>+D 
<a ,  Mvp> 


M  v_ 


(2-29) 
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2.2  --  Continued. 


where  M'  represents  the  row  vector  made  up  of  the  elements  of  the 
third  row  of  the  matrix  M  .  From  (2-29)  , 


<a,  MVp> 

<a, v'>+D 
'  o 


f' 


<a'V+D  Mv  > 

— -  P 

<"a,  M  v  > 

P 


(2-30) 


Substituting  (2-30)  into  (2-28) , 


f'  NT  1  (<a,  M  v  >  (v  -v-*)  -  (<a,v  >+D)  Mv  ) 

v'  =  - - E - 2— - - 2 - E  '  (2-31) 

p  <M'“  ,<a,  Mvp>(vQ-v')  -  (<avvQ>+D)  Mvp>. 

which  can  also  be  written  as 


f^<M'  ,  <a ,  Mv  >  (v  -v')  -  (<a,v  >+D)Mv  > 

x;  =  - - - ° - 2 - 2-  (2-32) 

p  <M^  ,<a,  M  vp>  (vQ-v^)  -  (<a,vQ>+D)  Mvp> 

f"<M^-1,<a,  Mv  >(v  -v')  -  (<a,v  >+D)  Mv  > 

z'  =  - 4 - 2 - £_^ - 2 - E_  (2-33) 

p  <M'  ,  <a ,  M  v  >  ( v  -v"* )  -  (<afv  >+D)Mv  > 

Equations  (2-32)  and  2-33)  are  the  two  camera  perspective 

equations.  They  are  fractional  linear  in  form  (i.e.,  the  numerator 

and  denominator  are  linear  functions  of  x  and  z  ) .  The  equations 

p  P 

are  valid  as  long  as  the  plane  does  not  contain  either  camera. 
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3. _ IMAGE  MODEL. 

Black  and  white  aerial  photography  was  taken  with  a  12-inch 
focal  length  RC-10  camera  at  3000  to  3500  feet  altitude  and  depression 
angles  of  50  to  6C  degrees  from  horizontal.  The  imagery  was  digitized 
with  10-bit  precision  and  10  to  30  pm  pixel  size  with  a  PDS-1010 
scanning  microdensitometer. 

The  image  model  is  the  ancillary  information  used  to  fully 
define  the  relationship  of  a  digital  image  to  a  spatial  coordinate 
system.  This  information  consists  of  the  camera  model  (the  position, 
orientation,  and  focal  length  of  the  camera)  and  the  scanning  param- 
meters  (the  position  of  the  microdensitometer  origin  with  respect  to 
the  image  center  and  the  pixel  spacing) .  This  section  discusses  the 
acquisition  of  the  camera  model  and  the  scanning  parameters. 

3 . 1 _ Camera  Model  Derivation  and  Extraction. 

The  problem  of  identifying  the  position  and  orientation  of 
a  camera  from  its  photography  is  known  as  resectioning.  A  number  of 
techniques  for  resectioning  are  described  in  the  literature  of  photo¬ 
gramme  try  . 


General  solutions  have  been  found  that  require  iterative 

12  3  .  . 

techniques.  '  '  They  require  an  initial  approximation  to  the  three 

spatial  coordinates  and  three  orientation  angles  of  the  camera.  At 

least  three  pairs  of  conjugate  picture  and  spatial  control  points  are 

required  and  used  with  the  perspective  equations  in  a  least  squares 

procedure  to  find  corrections  to  the  initial  approximations. 

4  5 

Closed  form  solutions  have  also  been  developed.  '  However, 
they  make  the  rather  limiting  assumption  of  nearly  downlooking  views. 
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3  .  1 _ --  Continued. 

We  have  developed  an  approach  that  is  straight-forward  and 
simple  in  its  application,  but  requires  a  rectangular  parallelepiped 
structure  in  the  photograph  located  near  the  optical  axis.  It 
requires  that  one  pair  of  conjugate  control  points  be  measured  in 
the  image  and  on  the  ground,  in  addition  to  the  coordinate  system 
origin . 


Consider  Figure  3-1,  which  shows  an  image  of  a  rectangular 
parallelepiped  structure  located  on  horizontal  terrain.  When  projected 
into  the  image,  the  spatial  corner  point  labelled  p  ,  is  located  at  the 
optical  center  of  the  photograph  which  is  the  origin  of  the  picture 
coordinate  system.  The  origin  of  the  spatial  coordinate  system  is 
centered  at  the  corner  of  the  structure,  with  its  axes  parallel  to  the 
sides  as  shown.  If  the  angles  ft  and  Y  and  the  slope  of  the  vertical 
edge  defined  by  pQp3  are  measured  in  the  photograph,  then  knowledge  of 
these  parameters  along  with  the  perspective  equations  applied  to  the 
spatial  points  p^ ,  p2  and  p^  is  sufficient  to  solve  for  the  pan,  tilt, 
and  roll  angles  0,<t>,£  describing  the  camera  orientation  as  follows. 

Consider  the  projection  of  the  spatial  points  p^,  p2  and  p^ 

into  the  image.  From  the  perspective  equations  (2-9)  and  (2-10) , 

these  image  points  can  be  expressed  as  p  .  =  (x  . ,z  .)  ,  where 

pi  P  ^  P * 

i  =  1,2,3  (3-1) 


In  these  expressions  C.  =  f/<@,,v. -v  >,  and  v.  is  the  vector  from  the 
origin  to  the  spatial  points  p^  Since  the  spatial  origin  is  at  pQ , 
the  vector  from  the  origin  to  the  camera  position,  v  ,  will  be  parallel 
to  the  unit  vector  §2.  Thus 


x  .  =  C . <£. , v. -v  > 
pi  1  1'  1  o 


2pi  -  WVV 
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3.1  --  Continued. 


<S1'V  =  <§3'V  =  ° 


(3-2) 


If  p  .  are  defined  to  be  vectors  from  the  picture  origin  to 
P  ^ 

the  picture  points  p  then  using  equations  (3-1)  and  (3-2)  one 
obtains 


rx 

c . 

<§. , v . > 

pi 

z 

L  piJ 

i 

1 '  l 

i  =  1,2,3 


( 3-3: 


n 

"0" 

-  O' 

Choosing  v.  = 

i - 

o  o 

II 

(N 

1> 

1 

.0. 

and  v3  = 

0 

r  1- 

leads  to 


Pi 


p  0  =  C 
*p2  j 


and 


p3 


r  cos; 

cosQ 

+  sin; 

sin<f> 

sin©  1 

[-sin; 

COS0 

+  cos; 

sintj) 

sin0j 

["  cos; 

sinB 

-  sin; 

sin$ 

COS0  1 

[-sin; 

sin0 

-  cos; 

sin<$> 

COS0  J 

Jsin; 

COS()>" 

Loos; 

cost)). 

(3-4) 


(3-5) 


(3-61 


From  equation  (3-6) 


tan 


-1  /xp3\ 

l  z  -) 
\  p3/ 


-1  /sin£\ 
tan  \cos;/  = 


(3-7) 
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3 . 1 _ --  Continued. 

This  expression  indicates  that  the  slope  of  the  vertical 
edge  of  the  building  PqP^,  as  measured  in  the  image,  identifies  the 
roll  angle,  t, ,  of  the  camera. 

The  remaining  orientation  angles  can  be  obtained  from  the 
angles  SI  and  H*  measured  in  the  photograph.  These  angles  can  be 
expressed  from  the  definition  of  the  inner  product  as 


cosS] 


<Ppl  *  Pp3> 

IPplII  Pp3l 


and 


cos? 


<PP2  '  PP3> 

I  Pp2  I  I  JW 


(3-8) 


(3-9) 


Substituting  equations  (3-4),  (3-5),  and  (3-6)  into  equations  (3-8) 
and  (3-9) ,  one  obtains 


cosft 


-sin<j>  sin8 _ _ 

(cos^Q  +  sin%  sinO)^ 


(3-10) 


I 

I 

I 

I 

I 

r  i«h 


and 


cos^  =  ?i.nf  ?os9 - 5 - - - -  =  <  (3-11) 

(sin^6  +  sin  <J>  cos  6)x/ 

Squaring  both  sides  of  equations  (3-10)  and  (3-11)  and  solving  for  the 
numerator  then  leads  to 
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—  Continued. 


2  2  2  2  2  2  2 
sin  v  sin  9  =  cos  0  +  sin  <J>  sin  6 


2  2  2  2  2  2  2 
sin  $  cos  G  =  <2  sin  0  +  <2  sin  <J>  cos  0 


From  equation  (3-12)  one  obtains 


2  2 

2  K.COt  0 

sin  <J>  =  - = - 

1-K1 


Similarly  from  equation  (3-13)  one  obtains 


(3-12) 


(3-13; 


(3-14) 


sm  4> 


2 .  2, 

K^tan  0 


(3-15) 


Equating  (3-14)  and  (3-15)  and  solving  for  9, 


tan  0  =  -4 


2  2 

cos  ft  sin  Y 

2  2 

cos  ¥  sin  ft 


(3-16) 


-  -  tan 


1  rtsal 

|_tanft  J 


(3-17) 
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3 . 1 _ —  Continued. 

where  the  negative  root  is  chosen  to  be  consistent  with  the  definition 
of  the  pan  angle. 

If  equation  (3-16)  is  substituted  into  the  square  of  equation 
(3-15)  one  obtains 


sm 


tan  Q  tan  4* 


or 


=  -sin 


h/2 


tanQ  tanV 


(3-18) 


(3-19) 


Where  the  negative  root  is  chosen  to  be  consistent  with  the  definition 
of  the  tilt  angle.  (Note  that  tanft  and  tant  are  both  negative,  since 
for  the  orientation  of  the  building  shown,  both  angles  are  greater 
than  90°. ) 


The  above  formalism  was  developed  for  the  special  case  in 
which  the  optical  center  was  coincident  with  the  projection  of  the 
closest  corner  of  the  building  as  shown.  This  formalism  can  be 
extended  to  any  corner  of  the  building  for  which  the  three  inter¬ 
secting  edges  of  the  building  are  visible.  If  p^,  P2»  and  p^  are 
always  chosen  as  =  (±1,0,0)  P2  =  (0,±1,0)  and  =  (0,0, ±1)  then 
equations  (3-5),  (3-17),  and  (3-19)  become 


(3-20) 
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(3-21) 


(3-22) 


The  negative  sign  for  equation  (3-22)  is  always  correct,  since  the 
input  scenes  are  always  downlooking  and  thus  have  a  negative  tilt. 
The  sign  for  equation  (3-21)  must  be  determined  by  the  orientation 
of  the  projection  of  the  Y  spatial  axis  in  the  photograph.  If  it 
points  towards  the  left  side  of  the  photograph,  6  will  be  negative 
If  it  points  towards  the  right  side  of  the  photograph,  then  9  will 
be  positive. 


Once  the  camera  orientation  angles  have  been  identified, 
the  camera  position  can  be  obtained  from  an  application  of  the 
inverse  perspective  equations  with  at  least  one  pair  of  conjugate 
picture  and  spatial  coordinates.  The  picture  coordinates  can  be 
measured  either  directly  from  the  original  photograph,  or  in  the 
digital  image  and  converted  to  picture  coordinates  as  will  be 
described  in  the  following  section.  The  spatial  coordinates  are 
measured  on  a  map  and  are  generally  at  ground  level. 

Hr°i 

,  the  spatial  origin,  then  v^  =  0  , 

since  the  projection  of  the  spatial  origin  lies  at  the  optical  center. 
Substitution  of  these  values  into  the  inverse  perspective  equation, 
equation  (2-17),  leads  to 

vo  =  -Xf§2,  (3-23) 
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3.1  --  Continued. 


which  indeed  shows  that  vq  is  parallel  to  §2'  When  equation  (3-23) 
is  substituted  into  the  perspective  equations,  equations  (2-9)  and 
(2-10) ,  one  obtains 


A 


f<fe. 


,  v>  - 


X  <£_  , 
-  £  2. 


v> 


fx 


(3-24) 


and 


A 


f<S. 


,  v>  - 


Z  <g„,V> 

_£ _ _ 


f  z 


(3-25) 


The  parameter  A  can  be  obtained  by  substitution  of  at  least  one  pair 

-*•  T 

of  conjugate  spatial  coordinates,  v  =  (x,y,z),  and  picture  coordinates, 
Vp  =  (Xp,Zp),  into  equations  (3-24)  and  (3-25)  and  averaging  the 
results.  The  camera  location,  v  =  (x  ,y  ,z  )T,  then  can  be  determined 
by  substitution  of  this  value  for  A  into  equation  (3-23). 


If  the  spatial  origin  is  selected  at  ground  level,  then  the 
analysis  of  the  camera  position  follows  by  a  straight-forward  appli¬ 
cation  of  the  above,  namely  equations  (3-23),  (3-24),  and  (3-25).  If 
however,  the  spatial  origin  is  above  ground  level,  as  shown  in 
Figure  3-1,  then  the  coordinates  of  the  spatial  control  points  will 
be  at  values  below  z  =  0.  This  non-zero  offset  is  equal  to  the  height 
of  the  building  at  the  spatial  origin.  In  most  cases,  this  height  is 
not  available  on  the  map.  It  can  be  determined  in  at  least  two  ways. 
The  first  way  is  to  perform  shadow  mensuration,  provided  that  a 
clearly  visible  shadow  is  available  for  the  building  in  question. 

The  second  method  is  an  iterative  approach.  First,  assume  the  origin 
is  at  ground  level.  Obtain  the  camera  position  as  described  above. 
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3.1  --  Continued. 


Then  use  the  3-D  stick  figure  model  algorithm  (to  be  described  below) 
with  this  camera  model  to  find  the  height  of  the  building.  Iterate  this 
technique  to  the  desired  accuracy. 

3 . 2 _ Scanning  Parameters. 

In  addition  to  the  camera  parameters,  certain  scanning 
parameters  are  needed  to  relate  the  digital  image  coordinate  system 
measured  in  lines  and  samples  into  picture  coordinates  required  by 
the  perspective  equations.  These  parameters  are:  the  sample  spacing 
and  the  offsets  between  the  optical  center  and  the  upper  left  corner 
of  the  digital  image  subsection  scanned. 

Figure  3-2  depicts  a  piece  of  film,  a  small  section  of  which 
is  to  be  scanned  with  a  microdensitometer  and  converted  into  digital 
form.  There  are  three  basic  coordinate  systems  to  be  considered,  each 
of  which  is  shown  in  this  figure. 

The  first  coordinate  system  is  that  of  the  microdensitometer 

whose  coordinate  axes  are  labelled  X  and  Z  .  Note  that  the  direction 

s  s 

of  the  Xg  axis  is  opposite  to  the  standard  format.  The  units  of  this 
system  are  in  micrometers  and  the  origin  is  arbitrary. 

The  second  coordinate  system  is  the  picture  coordinate  system 

whose  axes  are  labelled  X  and  Z  .  It  also  is  measured  in  units  of 

P  P 

micrometers  and  its  origin  is  at  the  optical  center  of  the  image.  The 
perspective  equations  are  described  in  terms  of  this  coordinate  system. 

The  third  coordinate  system  is  that  of  the  digital  image. 

It  is  measured  in  pixels  along  the  line  (L)  and  sample  (S)  directions. 
Pixel  (1,1)  is  the  upper  left  hand  corner  of  the  digital  image. 
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3.2  —  Continued. 


When  a  picture  is  scanned  by  a  microdensitometer,  a  light 
spot  of  fixed  size  is  moved  across  the  film  at  fixed  intervals  or 
steps.  For  each  placement  of  the  spot,  the  light  intensity  transmitted 
(or  reflected)  is  coded  as  the  intensity  or  gray  level  of  the  image  at 
that  position.  The  step  size  or  interval  moved  between  measurements 
is  called  the  sample  spacing  (SS) .  The  number  of  measurements  in  the 
horizontal  directions  indicates  the  sample  dimension  in  pixels  and 
the  number  of  measurements  in  the  vertical  direction  identifies  the 
line  dimension  in  pixels  for  the  image. 

If  one  defines  the  upper  left  corner  of  the  digital  image 

(i.e.,  line  one  and  sample  one)  to  be  at  scanning  coordinates 

(x^,  zi) ,  and  if  one  also  defines  the  optical  center  (origin  of  the 

picture  coordinate  system)  to  be  at  scanning  coordinates  (xc,  zc) , 

then  the  xs  and  zg  offsets  between  these  two  positions  in  scanning 

coordinates  can  be  expressed  as  Ax  =  (x.  -  x  )  and  Az  =  (z.  -  z  ). 

sic  sic 

It  then  follows  that  the  relationship  between  the  digital  image  and 
picture  coordinate  system  can  be  expressed  as: 


x  =  -Ax  +  SS(S-l) 
P  s 

z  =  Az  -  SS(L-l) 
P  s 


or 


L 


S 


1  + 


+  Ax„ 
_ s 

SS 


(3-26) 

(3-27) 

(3-28) 
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4_. _ THREE  DIMENSIONAL  STICK  FIGURE  MODEL. 

A  photograph  is  a  two-dimensional  representation  of  the 
three-dimensional  world.  A  human  is  able  to  understand  and  interpret 
a  photograph  because  he  is  aware  of  the  3-D  model  behind  the  2-D 
projection.  Just  as  a  human  can  imagine  what  a  photograph  would  look 
like  from  another  camera  position,  computerized  image  processing 
techniques  can  synthesize  new  perspective  views  if  the  underlying 
3-D  model  is  known.  In  this  project  we  have  used  a  stick  figure 
model  comprising  the  edges  of  planar  surfaces  in  3-D  coordinates  as 
the  representation  of  the  spatial  relationships  within  the  scene. 

This  section  describes  the  operation  of  the  software  that 
extracts  and  manipulates  the  stick  figure  model  (Section  4.1)  and  the 
mathematical  details  of  the  program  (Section  4.2.). 

4 . 1 _ Operation. 

The  3-D  stick  figure  model  algorithm  is  an  interactive 
program  written  for  the  Interactive  Digital  Image  Manipulation 
System  (IDIMS) ,  which  allows  an  operator  to  identify  planar  surfaces 
for  transformation  from  one  perspective  view  to  another.  The  algorithm 
is  composed  of  two  basic  modules:  the  build  module  and  the  project 
module. 


The  build  module  allows  the  operator  to  construct  a  three- 
dimensional  stick  figure  representation  of  two  different  types  of 
features.  One  feature  (type  1)  is  a  simple  convex  planar  surface. 

The  other  feature  (type  2)  is  a  rectangular  parallelepiped  composed 
of  6  planar  surfaces. 

For  type  1  features,  the  basic  scheme  is  as  follows.  The 
operator  places  the  cursor  on  any  boundary  point  of  the  planar  surface 
and  the  line  and  sample  coordinates  of  the  point  are  automatically 
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4.1  --  Continued. 


extracted.  He  then  inputs  any  one  of  the  (x,  y,  or  z)  components  of 
the  spatial  coordinate  of  the  point.  The  inverse  perspective  equations 
are  then  used  to  compute  the  two  remaining  components  of  the  spatial 
coordinate.  Spatial  coordinates  are  likewise  calculated  for  at  least 
two  other  boundary  points  {specified  consistently  in  either  a  clock¬ 
wise  or  counterclockwise  order).  As  each  point  is  identified  it  is 
marked  on  the  input  image  and  a  line  is  drawn  from  one  point  to  the 
next  completely  outlining  the  surface. 

After  a  sufficient  number  of  points  have  been  identified, 
the  spatial  coordinates  are  used  to  compute  the  equation  of  the  best 
fit  plane  (in  three  dimensions)  representing  these  points.  A  least 
square  orthogonal  fit  method  is  employed.  This  method  identifies  the 
coefficients  of  the  planar  equation  to  within  a  multiplicative  factor 
of  ±1;  that  is,  it  does  not  identify  the  direction  of  the  normal  to 
the  plane. 


After  this  step  the  operator  specifies  a  visibility  code 
parameter  (+1,  0  or  -1) ,  which  serves  a  two-fold  purpose.  It  identi¬ 
fies  the  direction  of  the  normal  to  the  surface,  and  identifies 
whether  gray  level  information  is  available  in  the  input  image  {for 
transformation  of  the  surface  to  an  output  perspective) .  By  defini¬ 
tion  the  surface  normal  always  points  outward  from  a  feature  and 
generally  towards  some  observer  in  the  spatial  coordinate  system  who 
can  see  the  gray  level  information  on  the  face  of  the  plane.  In 
terms  of  the  visibility  code  parameter,  a  negative  value  indicates 
that  the  surface  points  away  from  the  camera  and  no  gray  level 
information  is  available,  (e.g.,  the  back  face  of  a  building). 

Another  input  view  is  therefore  required  for  this  surface.  On  the 
other  hand,  a  non-negative  value  indicates  that  the  surface  normal 
generally  points  towards  the  camera  and  gray  level  information  is 
visible.  A  positive  parameter  value  represents  an  unobstructed  view 
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4 . 1 _ --  Continued. 

of  the  surface  so  that  correct  gray  level  information  is  available. 

A  zero  parameter  value  indicates  that  there  is  gray  level  information 
available,  but  the  surface  is  partially  obstructed  by  another  feature. 
(This  will  lead  to  an  incorrect  representation  of  the  surface  in  the 
output  perspective  and  requires  another  input  view  or  modification  of 
the  gray  level  information  for  a  correct  representation.)  Once  the 
visibility  code  parameter  is  specified,  the  sign  of  each  coefficient 
of  the  equation  of  the  plane  is  corrected  to  be  consistent  with  the 
direction  of  the  normal  in  the  spatial  coordinate  system. 

The  final  step  is  to  push  the  spatial  coordinates  of  the 
boundary  points  back  into  the  plane.  This  is  done  by  dropping  a 
perpendicular  from  the  point  to  the  plane  and  finding  the  inter¬ 
section  point. 

For  type  2  features,  the  assumption  of  a  rectangular 
parallelepiped  allows  for  the  efficient  computation  of  the  six  sur¬ 
faces  by  identifying  any  pair  of  diagonally  opposite  corners  of  the 
feature.  Figure  4-1  shows  a  representation  of  a  type  2  feature.  The 
corners  are  always  identified  in  the  fashion  shown.  Thus,  for 
example,  the  dimensions  of  the  feature  are  completely  identified  if 
one  knows  the  spatial  coordinates  for  corners  1  and  4. 

The  basic  scheme  for  this  type  of  feature  is  as  follows. 

The  operator  selects  any  one  of  the  corners  1,  2,  3,  4,  5  or  7  as  a 
starting  point.  In  the  same  manner  as  for  type  1  features,  he 
positions  the  cursor  at  the  appropriate  line  and  sample  in  the  image 
and  specifies  any  one  of  the  (x,  y,  or  z)  components  of  the  coordinates 
for  that  point.  The  other  two  components  are  then  calculated.  Once 
the  first  corner  is  specified,  he  is  prompted  with  a  specific  sequence 
of  3  other  corners,  which  will  end  at  the  diagonally  opposite  corner. 
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4.1  --  Continued. 

For  example,  if  corner  7  was  initially  specified,  then  the  next 
corner  prompted  would  be  corner  2.  The  operator  then  positions  the 
cursor  at  the  appropriate  image  coordinate.  The  algorithm  automa¬ 
tically  assigns  the  Z  component  of  corner  2  equal  to  the  Z  component 
of  corner  1  and  computes  the  corresponding  X  and  Y  components.  The 
next  prompt  will  be  for  corner  3.  Once  again  the  operator  positions 
the  cursor  and  the  algorithm  assigns  independently  the  X  and  Y  com¬ 
ponents  of  corner  3  equal  to  the  corresponding  components  of  corner  2. 
For  each  of  these  components  the  remaining  two  spatial  components  are 
calculated  and  the  two  resulting  sets  are  then  averaged  to  obtain  the 
spatial  coordinates  of  corner  3.  The  last  corner  prompt  is  for 
corner  5.  The  operator  again  places  the  cursor  on  the  corresponding 
point  and  the  algorithm  assigns  the  Z  component  of  corner  5  equal  to 
the  Z  component  of  corner  3.  Next  the  coordinates  of  the  remaining 
corners  are  computed  assuming  a  rectangular  parallelepiped  configura¬ 
tion.  Finally  the  equations  for  five  surfaces  (the  base  is  excluded, 
since  it  can  never  be  observed)  are  calculated  using  the  least  squares 
orthogonal  fit  method,  the  normals  are  calculated  from  assigned 
visibility  code  values  for  each  surface  (the  two  back  surfaces  are 
automatically  assigned  a  visibility  code  value  of  -1) ,  and  the  out¬ 
line  of  each  surface  ir  drawn  on  the  input  image. 

The  project  module  allows  the  stick  figure  to  be  projected 
from  its  spatial  representation  to  any  two-dimensional  output  perspec¬ 
tive.  Projection  can  be  specified  for  single  points,  surfaces, 
features  or  the  composite  stick  figure.  The  spatial  coordinates  for 
a  given  surface  are  projected  one  at  a  time  using  the  perspective 
equations  to  the  output  view,  and  are  then  connected  by  straight 
lines  in  order  to  outline  the  surface. 


Z++.\  ..  "  r  —  -  — -  ~T  "  —  * 
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4.2  Mathematical  Derivation. 


The  five  key  components  to  the  stick  figure  model  algorithm 
can  be  summarized  as  follows: 

•  Extract  spatial  coordinates  for  boundary  or  corner 
points . 

•  Calculate  the  least  square  fit  planar  equation. 

•  Correct  the  planar  equation  for  the  surface  normal. 

•  Push  the  boundary  points  into  the  plane. 

•  Project  the  3-D  stick  figure  model  to  2-D  output 
perspective . 

The  mathematical  analysis  of  each  of  these  five  key  components  is  now 
described  below. 

4.2.1 _ Extract  Spatial  Coordinates. 

The  extraction  of  spatial  coordinates  (x,  y,  z)  corresponding 
to  a  given  set  of  digital  image  coordinates  first  requires  the  con¬ 
version  from  a  set  of  digital  image  coordinates  to  a  set  of  picture 
coordinates,  followed  by  an  application  of  the  inverse  perspective 
equations.  The  former  can  be  obtained  by  a  straightforward  applica¬ 
tion  of  equations  (3-26)  and  (3-27).  The  latter  is  more  complex, 
because  only  two  picture  coordinates  are  input,  and  there  are  three 
inverse  perspective  equations  (2-20)  ,  (2-21)  and  (2-22)  for  each  of 
the  spatial  coordinates.  Thus,  in  order  to  obtain  the  inverse  perspec¬ 
tive  transformation,  at  least  one  spatial  coordinate  must  also  be 
specified.  With  this  in  mind  we  shall  treat  the  three  cases  separately, 
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4.2.1 _ --  Continued. 

each  one  pertaining  to  a  different  initially  specified  component  o; 
the  spatial  coordinate  system. 

Case  A 


If  z  is  initially  specified,  then  one  can  solve  for  '■  from 
equation  (2-22).  Thus 


z  -  z 


A  = 


M,  ,  X  +  M-._z  +  M-,  f 

31  p  32  p  33 


Substituting  this  equation  back  into  equations  20  and  21  to  eliminate 
the  free  parameter  leads  to 


x  =  xq  +  (z-zQ) 


y  =  yQ  +  (z-zQ) 


Mllxp  +  M12Zp  +  M13f 


M31Xp  +  M32Zp  +  M33f 


Case  B 


(4-1) 


(4-2) 


In  a  similar  fashion,  if  y  is  specified  initially,  then 
one  obtains 


x  = 


xo  + 


Mllxp  +  M12Zp  +  M13f 
M21XP  +  M22Zp  +  M2  3f 


(4-3) 


z  = 


zo  +  (^-^o) 


M31Xp  +  M32Zp  +  M33f 
M21xp  +  M22zp  +  M23f 


(4-4) 
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4.2.1  —  Continued. 


Case  C 


obtains 


In  the  same  manner,  if  x  is  specified  initially,  then  one 


y  =  yG  +  <x-*0> 


z  =  z  +  (x-x  ) 
o  o 


x  +  M00z^  +  M~-.f 
21  p  22  p  23 


M..x  +  M. 0z^  +  M. ,f 
11  p  12  p  13 


(4-5) 


(4-6) 


These  three  pairs  of  equations  completely  describe  the 
inverse  perspective  equations  as  applied  to  this  problem. 


4.2.2 _ Least  Square  Orthogonal  Planar  Fit. 

While  the  method  of  fitting  curves  to  data  in  the  least 
squares  sense  is  a  very  familiar  one,  most  published  analyses  and 
applications  define  only  one  component  of  the  distance  of  each  point 
from  the  curve  as  the  error  whose  sum  of  squares  is  to  be  minimized. 
In  effect,  one  component  of  the  distance  is  defined  as  the  dependent 
one,  while  the  others  are  defined  as  independent.  The  choice  of 
dependent  variable  affects  the  solution  obtained.  For  example  fit¬ 
ting  a  straight  line  (Ax  +  By  +C  =  0)  to  a  set  of  points  produces 
one  solution  if  one  minimizes  the  sum  of  the  squares  of  the  errors 
Ay  and  another  solution  by  minimizing  the  Ax  sum  of  squares.  The 
following  analysis  presents  a  method  for  obtaining  the  best  planar 
fit  to  a  set  of  points,  where  best  is  defined  as  minimizing  the  sum 
of  the  squares  of  the  orthogonal  errors. 
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4.2.2  --  Continued. 

Let  the  general  equation  for  the  plane  that  is  to  be  fit  to 
the  points  be: 

Ax  +  By  +  Cz  +  D  =  0  (4-7) 

Then,  the  orthogonal  distance  of  a  point  '  Y '  zj^  from  this  plane 
is  given  by: 


(Ax.  +  By .  +  Cz .  +  D) 
R  =  i  1 

1  (A2  +  B2  +  C2)  1/12 


(4-8) 


Thus,  the  function  that  is  to  be  minimized  is: 


N  N 

Y  R-  =  V  (Axi  +  By  ^  +  Czi  + 

Z>  /  - 2 - 5 - 5~ 

i=l  PT  +  B +  c 


(4-9) 


Using  the  standard  technique  of  taking  first  partial  derivatives  of 

(4-9)  with  respect  to  the  coefficients  (A,B,C,D)  we  have  (after 

2  2  2 

dividing  through  by  2 (A  +B  +C  ) ) : 


“  (Axi+Byi+Czi+D)xi  -  A  (Ax^By^Cz i+D)  2  1 

2,  - =5 — * — * -  =  o 


A  +B  +C 


(4-10) 


2  (Axi+Byi+Czi+D)yi  -  B ( Ax^+By^+Cz^+D) 2 


A2+B2+C‘ 


(4-11) 


2,  (Ax.+By.+Cz.+D)z .  -  C (Ax . +By . +Cz . +D) 2 
v  i  ll  l  i  ll  _n 

z,  5  2  2  =  0 

i=l  L  A  +B*+(T 


(4-12) 


4-9 


ESL-PR3  39 


4.2.2  --  Continued. 


£  (Axi+Byi+Czi+D)  =  0 

i=l 


(4-13) 


Then,  define  a  new  parameter,  y,  as: 


«  (Axi+By . +Czi+D) 2 

^  a2+b2+c2 


(4-14) 


Equations  (4-10)  through  (4-12)  now  become: 


N  -  N  N  N 

a  (  £  x .  -  Y )  +  B  Y  x  .  y  .  +  c£x.z,  +  D  x.  =  0 

i=l  i=l  1  1  i=l  1  i=l  1 


(4-15) 


N  N  N  N 

A  Z  +  B(  X  yi~  Y)  +  C  £  +  D  £  y  .  =  0 

i=l  i=l  1  i=l  x  x  i=l 


(4-16) 


A  £  x  .  z  .  +  B  £  y  .  z  .  +  C  (  Y  z  .  -  y)  +  D  £  z  . 
i=l  1  i=l  1  1  i=l  1  i=l  1 


(4-i7: 


and  equation  (4-13)  can  be  solved  for  D  as: 


D  =  -Ax-By-Cz  , 


(4-18) 


-  -  T 

where  the  bar  over  the  variable  implies  the  mean  value  (eg.,  x  h  Nix.) 

i  =  l  1 


Substituting  (4-18)  into  the  previous  three  equations  yields  (in 
matrix  form) : 
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4.2.2  --  Continued. 


- 

-  - 

- 

N 

- 

-2  N 

Nx  ~Y )  (  £  x.y.-  Nxy) 

i=l  1  1 

N 

(  £  xiZ 
i=l  1  1 

-  Nxz) 

A 

0 

N 

N 

N 

(  ^,XiYi' 

-  Nxy)  (  £  yZ  -  NyZ  -y) 

(  £  Yiz± 

-  Nyz) 

B 

= 

0 

1=1 

i=l  1 

i=l  1  1 

N 

N 

N  - 

(  y  x .  z . 

•  l  i 

-  Nxz )  ( ^  y.z.  -  Nyz ) 

<  E  - 

Nz2  -y) 

C 

0 

1=1 

i=l  1  1 

i=l  1 

(4-19) 


But,  note 


N  N  „  N  ~ 

that  J  xf  -  Nx2  =  Y  (x.  -  x)2  =  V  x2 

1=1  1  i=l  1  i=l  1 


and  £  x . 


N 


N 


.£  -  Nxy  =  £  -  *)  (y.  -  y)  =  £  X.y. 

i=l  i=l  1  i=l  1 

Thus,  (4-19)  may  be  rewritten  as: 


or 


N  N 

(  £  xi  -y)  (  £  xiyi) 

i=l  1  i=l  1  1 

1ST 

(  l^z.) 

i=l  1  1 

r  i 

A 

0 

N  N 

(  Z^iYi)  <  -Y) 

N 

<  £  Yi2i) 
i=l  1  1 

B 

= 

0 

N  N 

(£Vzi)  ( 

_  i =1  11  i=l  1  1 

£  22i  -Y) 
i=i  1 

C 

0 

f‘11  “12  a13~|  fAl 

‘“ll-Y) 

a12 

a13 

1 

a21  a22  a23 
a31  a 32  a33 


21 

‘31 


(a22“Y)  a 


l32 


23 


(a33-y) 


(4-20) 


(4-21) 
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4.2.2  --  Continued. 


Thus,  in  order  for  equation  (4-19)  to  have  a  non-trivial  solution 
(A/0,  B/0,  C/0),  the  matrix  must  be  singular  (i.e.,  the  determinant 
must  =  0) .  This  can  be  accomplished  through  a  suitable  choice  of  > , 
which  is  an  eigenvalue  of  the  matrix.  Recall  that  y  was  defined  in 
terms  of  the  coefficients  A,  B,  C,  and  D,  and  is  in  fact,  the  very 
function  that  is  to  be  minimized.  The  determinant  of  the  matrix 
in  (4-20)  is  a  third  order  polynomial  in  y: 

Y3  +  PY2  +  qY  +  r  =  0  (4-22) 


where  p 

q 

r 


-(°‘ll  +  a22  +  0t23) 

2  2  2 

alla22  +  alla33  +  a22a33  "  a12  "  a13  '  a23 

alla23  +  a22a13  +  Cx33a12  ~2  a12a13a23  “  alla22a33 


Equation  (4-22)  has  three  roots,  all  of  which  must  be  real,  since  they 
are  eigenvalues  of  the  symmetric  matrix  in  (4-20).  The  parameter  y, 
as  defined  by  (4-14)  is  the  function  that  is  to  be  minimized,  so  the 
only  valid  (non-extraneous)  solution  is  the  smallest  non-negative 
eigenvalue.  Note  that  if  r  =  0,  then  the  solution  desired  is  y  =  0. 


If  r  /  0,  then  the  three  roots  of  (4-22)  may  be  found  from: 


Y1 

Y2 

Y3 


2  v/-a  cos(£/3)  -  p/3 
3 

2  cos  ( 3/3  +  120°)  -  p/3 

I 

2  J^a  cos  ( 3/3  +  240°)  -  p/3 


(4-23) 
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4.2.2  --  Continued. 


where : 

a  =  q  -  p2/3  (4-24) 

and , 

p  =  principal  value  of  cos 
b  =  r  -  pq/3  +  (2/27)p3 

Once  the  smallest  non-negative  eigenvalue  is  found,  it  is  used  to 
determine  the  elements  of  the  matrix  in  (4-21).  Then,  the 

coefficients  of  the  desired  equation  (4-7)  may  be  found  from: 


A  =  k(ai25j3 

-  5j2Si3> 

B  =  k(ajl5i3 

-  aiiaj3> 

(4-27) 

C  =  k(ai]Ldj2 

-  ajl5i2> 

Any  two  rows,  i  and  j  from  (4-21)  are  suitable  provided  that  the 

choice  does  not  result  in  all  parenthetical  factors  =0.  In  the 

above  expressions  k  is  a  proportionality  constant  that  is  arbitrarily 

222  1/2 

chosen  to  be  k  =  ± (A  +B  +C  _)  in  order  to  yield  conveniently  sized 
coefficients  for  equation  (4-7) .  Finally  C  can  then  be  uniquely 
determined  from  equation  (4-18)  : 

D  =  -  Ax  -  By  -  Cz  (4-28) 

Thus,  the  least  squares  solution  for  orthogonal  errors  has  been 
determined  for  a  linear  function  of  the  variables  x^ ,  y^,  and  z^. 


[rreTr,] 


(4-25) 


;  4- 26 ) 
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4.2.3  _ Direction  of  the  Surface  Normal. 

In  the  previous  section,  the  coefficients  describing  the 
least  squares  method  for  fitting  plane  were  described.  The  first 
three  components  (A,B,C)  also  comprise  the  vector  (in  spatial  coordi¬ 
nates)  ,  which  is  normal  to  the  plane.  The  inclusion  of  a  dual-valued 
scaling  constant  k  allows  the  freedom  to  pick  one  of  two  opposing 
directions  for  this  normal,  so  that  it  may  extend  from  the  side  of 
the  plane  that  contains  gray  level  information.  The  selection  of  the 
appropriate  sign  to  use  is  determined  from  the  visibility  code 
parameter  in  the  following  manner. 

Recall  that  a  non-negative  visibility  code  indicates  that 
the  important  gray  level  detail  of  a  surface  should  be  visible  to  the 
camera  and  that  a  negative  visibility  code  indicates  that  the  gray 
level  detail  is  not  visible.  Now,  since  the  feature  is  defined  in 
the  input  scene  (it  is  not  behind  the  camera) ,  a  sufficient  test  to 
use  to  identify  the  sign  for  the  surface  coefficients  can  be  derived 
using  the  visibility  code.  Start  with  k>0.  If  the  visibility  code 
is  non-negative,  then  the  camera  must  be  on  the  same  side  of  the 
surface  as  the  gray  level  detail.  If  the  position  of  the  camera 
<xo'yo,Zo>  is  inserted  into  the  expression  for  the  plane,  Ax  +  By  +  Cz 
+  D,  then  a  positive  number  should  result.  If  it  does  not,  then 
choose  k<0.  Similarly,  if  the  visibility  code  is  negative,  then  the 
camera  is  on  the  back  side  of  the  plane,  i.e.,  the  opposite  side  from 
that  with  the  gray  level  information.  In  this  case,  if  the  camera 
position  is  inserted  into  the  expression  for  the  plane,  a  negative 
number  should  result.  If  it  does  not,  then  choose  k<0. 

4.2.4  _ Push  Boundary  Points  into  the  Plane. 

The  spatial  boundary  points  used  to  extract  the  least  square 
orthogonal  fit  plane  do  not  necessarily  lie  in  the  plane.  In  order  to 
ensure  a  consistent  outline  of  the  surface  in  the  output  perspective, 
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4.2.4 _ --  Continued. 

it  is  necessary  to  push  these  points  back  into  the  plane.  This  is 
done  by  dropping  a  perpendicular  line  from  each  point  to  the  plane 
and  finding  the  intersection  of  this  perpendicular  line  with  the  plane. 

Consider  a  point  p^  =  (x^,y^,z^)  and  a  plane  Ax  +  By  +  Cz  + 

D  =  0.  A  line  that  passes  through  this  point  that  is  perpendicular 
to  the  plane  (parallel  to  the  normal  vector  fi  =  Ai  +  Bj  +  CR)  can  be 
expressed  as 


(4-29) 


If  the  point  of  intersection  of  this  line  with  the  plane  is  called 
pQ  =  (xo,yo,zo)  then,  point  pQ  must  satisfy  equation  (4-29)  ,  so  that 


xo'xl 


B 


Zo’Zl 


(4-30] 


Point  pQ  must  also  satisfy  the  planar  equation,  so  that 


Ax_  +  By  +  Cz„  +  D  =0 
o  o  o 


(4-31) 


One  may  now  solve  for  (ByQ)  and  for  (Czq)  in  terms  of  xQ  using 
equation  (4-30) . 


By0  =  B  (xo_xi>  + 
A 


(4-32) 


C  (xQ-x1)  +  Cz1 


(4-33) 
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4.2.4  --  Continued. 


Substitution  of  equations  (4-32)  and  (4-33)  into  equation  (4-31) 
leads  to 


x  = 
o 


-AD  +  (B2+C2)x1  -  ABy1  -  ACz1 


A  +B  +C 


(4-34) 


In  a  similar  manner  one  also  obtains 


yo  = 


-BD  +  (A2+C2)y1  -  BAx1  -  BCz1 

2  2  2 
A  +B  +C 


(4-35) 


2  = 
O 


-CD  +  (A2+B2)Zl  -  CAx1  -  CBy] 


[4-36  ) 


4.2.5 _ 3-D  to  2-D  Projection. 

The  projection  of  a  set  of  spatial  points  to  a  given  output 
perspective  view  is  described  by  the  perspective  equations  (2-9)  and 
(2-10) . 
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5.  GRAY  LEVEL  TRANSFORMATION. 


Gray  level  transformation  is  the  process  whereby  the  pixel 
by  pixel  gray  level  or  intensity  information  is  transferred  from  the 
original  digital  imagery  to  the  output  perspective  view.  A  few  papers 
have  been  published  wherein  gray  level  texture  prototypes  are  trans¬ 
ferred  via  perspective  transforms  to  generate  synthetic  images6,7, 
but  the  specific  problem  of  transferring  actual  gray  level  information 
for  real  scenes  has  not  been  previously  addressed. 

For  each  planar  surface  imaged,  the  input  and  output  image 
models  and  the  planar  surface  equation  in  the  stick  figure  model  are 
used  to  define  the  mathematical  transformation  (Section  5.1).  All  of 
the  transformed  surfaces  making  up  the  output  image  are  combined  in 
the  proper  relationship  to  one  another  (Section  5.2). 

5 . 1 _ Generation  and  Application  of  the  Gray  Level  Transformation. 

Section  2.2  explained  that  the  two  camera  perspective 
equations  depend  on  information  about  the  scene  being  imaged  as  well 
as  the  image  models  for  the  two  cameras.  We  have  made  the  assumption 
that  the  urban  scenes  we  work  with  can  be  modeled  as  a  collection  of 
planar  surfaces.  The  stick  figure  model  discussed  in  Section  4  con¬ 
tains  the  information  that  defines  each  planar  surface:  the  equation 
of  the  plane  and  the  vertices  that  define  the  plane's  polygonal 
boundary. 


The  stick  figure  model  information  for  a  given  plane  is  used 
to  supply  the  parameters  relating  the  input  and  output  image  coordinate 
systems  in  equations  (2-32)  and  (2-33) .  The  auxiliary  relationships 
(equations  (3-36)  and  (3-27))  between  image  coordinates  and  pixels  are 
used  so  that  the  transformation  between  the  input  and  output  coordi¬ 
nates  can  be  computed: 
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5.1  —  Continued. 


P1S'  +  p2l'  +  p3 

Q^S '  +  Q2L "  +  Q3 

R^S  '  +  R2L'  +  R3 

Q1S'  +  Q2L"  +  Q3 


(5-1) 


(5-2) 


where 


S  =  sample  number  in  input  image 
L  =  line  number  in  input  image 
S "  =  sample  number  in  output  image 
L'  =  line  number  in  output  image 

P.  ,  Qi#  R.^  =  parameters  that  depend  upon  the  image  models  and 
plane  equation  but  not  upon  S,  L,  S'  and  L' . 

The  fact  that  the  parameters  P^,  ,  and  do  not  depend  upon  the 

pixel  locations  means  that  equations  (5-1)  and  (5-2) ,  which  must  be 
evaluated  at  each  pixel,  are  quick  computationally. 

The  spatial  coordinates  of  the  planar  surface  vertices  are 
projected  to  the  output  image  coordinate  system  using  equations 
(2-9)  and  (2-10) ,  and  converted  to  line  and  sample  by  equations 
(3-28)  and  (3-29).  Equations  (5-1)  and  (5-2)  are  evaluated  at  each 
output  image  pixel  within  the  polygon  defined  by  the  projected  vertices 
and  represent  the  input  image  pixel  location  from  which  the  gray  level 
is  to  be  taken.  In  general,  L  and  S  fall  between  pixels,  so  bilinear 
interpolation  is  done  to  get  the  gray  level: 
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5 . 1 _ —  Continued. 

Gray  level  =  (1-Sp)  [  ( 1-Lp)  I  (Lj. ,  S  p )  +  LpI  (Lj+1 ,  SJ)  ] 
+Sp[ (1-Lf)J(Li,Si+1)  +  LpJ(LI+l,SI+l) ] 

where 


S  =  the  integer  part  of  S 

Sp  =  the  fractional  part  of  S 

Lj  =  the  integer  part  of  L 

Lp  =  the  fractional  part  of  L 

I  =  the  gray  levels  of  the  input  image. 

Evaluating  equations  (5-1) ,  (5-2)  ,  and  (5-3)  for  each  pixel 
of  the  output  image  that  falls  within  the  polygon  defined  by  the 
projected  vertices  completes  the  gray  level  transformation  for  the 
planar  surface.  This  process  is  repeated  for  every  plane  that  is 
contained  in  the  output  image. 


Combining  the  Transformed  Planes. 


The  planes  that  make  up  the  input  image,  after  gray  level 
transformation,  make  up  the  output  image.  Two  significant  processes 
must  be  performed:  missing  surface  addition  and  hidden  surface 
removal.  The  former  problem,  filling  in  the  areas  that  can  be  seen 
from  the  output  perspective  but  not  from  the  input  perspective,  has 
not  been  addressed  yet  in  this  study. 


The  latter  problem  consists  of  removing  the  surfaces  or 
parts  of  surfaces  that  fall  behind  other  surfaces  as  seen  from  the 
output  perspective.  A  related  problem,  removal  of  the  hidden  surfaces 
in  graphics  manipulations,  has  received  substantial  attention  in 

Q 

recent  years  . 
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5 . 2  _ --  Continued. 

These  techniques  are  not  completely  general  in  their  ability 
to  solve  the  hidden  surface  problem  and  can  involve  very  intricate 
logical  tests.  Up  to  this  time,  an  essentially  manual  approach  has 
been  used  in  this  project.  The  analyst  inspects  the  input  image  and 
the  stick  figure  model  to  determine  which  surfaces  should  have  priority 
over  the  other  surfaces  on  the  basis  of  the  proximity  to  the  output 
camera.  The  analyst  orders  the  surfaces  so  that  the  later  ones  are 
closer  to  the  camera.  The  surfaces  are  transformed  in  this  order  and 
written  to  a  single  output  image  file.  Each  surface  is  written  over 
any  surfaces  that  may  have  been  written  previously  at  the  same  loca¬ 
tion  on  a  pixel-by-pixel  basis.  Thus,  if  the  analyst  orders  the 
surfaces  properly,  the  hidden  surface  problem  is  handled  by  an  over¬ 
lay  process. 

Work  is  currently  being  done  to  make  the  hidden  surface 
removal  process  more  automatic.  The  concept  being  implemented  includes 
the  following  steps  to  eliminate  entire  surfaces  from  transformation: 

•  A  surface  will  be  eliminated  if  its  boundary  falls 
outside  the  output  image  (past  one  of  the  edges) . 

•  A  surface  will  be  eliminated  if  its  boundary  points 
are  behind  the  output  camera. 

•  A  surface  will  be  eliminated  if  the  output  camera  is 
on  the  wrong  side  of  the  surface  to  see  the  gray  level 
information. 

In  addition,  a  pixel-by-pixel  test  will  be  done  to  determine  which  of 
several  competing  surfaces  should  provide  the  gray  level  information 
at  each  point  of  the  output  image.  The  distance  from  the  camera  to 
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5 . 2 _ --  Continued. 

the  surfaces  (along  the  ray  through  the  pixel)  will  be  computed.  The 
closest  surface  to  the  camera  at  each  pixel  will  provide  the  gray 
level.  These  automatic  tests  should  be  sufficient  to  supplant  the 
manual  techniques,  thereby  reducing  the  time  spent  on  the  process 
and  eliminating  errors. 
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6_. _ EXAMPLES. 

6 . 1  _ Original  Photography. 

The  aerial  photography  was  taken  with  an  RC-10  camera  with 
a  12-inch  focal  length  at  approximately  3000  foot  altitude.  Nine-inch 
format  Plus-X  aerographic  film  was  used. 

Figures  6-1  and  6-2  are  examples  of  the  original  imagery. 

The  oblique  photographs,  such  as  Figure  6-1,  were  used  to  produce 
the  output  imagery.  The  vertical  photographs,  such  as  Figure  6-2, 
provided  some  location  information,  although  the  U.S.G.S.  7  1/2- 
minute  quadrangle  maps  supplied  most  of  this  data. 

6 . 2  _ Stick  Figure  Models. 

The  stick  figure  models  were  generated  using  the  computer 
program  described  in  Section  4,  which  allows  a  user  to  interactively 
select  image  points  to  define  a  stick  figure.  Figure  6-3  shows  a 
digitized  image  and  the  stick  figure  built  for  it.  Note  that  the 
automobiles  have  been  edited  out  of  the  scene  in  Figure  6-3  (b). 

There  are  five  surfaces  in  the  stick  figure  model: 

a.  The  sunny  side  of  the  building 

b.  The  shady  side 

c.  The  roof 

d.  The  east-west  street 

e.  The  north-south  street. 

Table  6-1  shows  the  equations  and  boundaries  of  the  five  planes. 
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b.  Stick  figure  model 
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Table  6-1.  Stick  Figure  Parameters 


Surface  Equation*  Boundary  points* 

x  y  z 


Sunny  side 

-.05x-l.00y-.02z  =  410.00  a. 

-807.7 

-404.6 

-77.1 

b. 

-807.1 

-405.8 

-12.3 

c . 

-713.2 

-406.2 

-12.3 

d. 

-713.1 

-405.0 

-77.1 

Shady  side 

-l.OOx  +  .01z  =  807.28 

a . 

-807.8 

-404.6 

-77.1 

b. 

-807.1 

-405.8 

-12.3 

c . 

-807.1 

-346.4 

-12.3 

d. 

-807  .8 

-345.7 

-77.1 

Roof 

z  =  -12.3 

a . 

-807.0 

-346.2 

-12.3 

b. 

-807.2 

-405.3 

-12.3 

c . 

-713.2 

-406.4 

-12.3 

d. 

-714.0 

-347.4 

-12.3 

East-west  street 

z  =  -77.10 

a.  -808.3 

-243.0 

-77.1 

b.  -807.9 

-431.6 

-77.1 

c.  -891.7 

-375.7 

-77.1 

North-south  street 

z  =  -77.10 

a . 

-848.8 

-404.4 

-77.1 

b. 

-690.8 

-405.1 

-77.1 

c . 

-739.2 

-477.4 

-77.1 

♦Units  are  feet.  The  x-axis  is  parallel  to  the  sunny  side,  the  y-axis  is  parallel 
to  the  shady  side,  and  the  z-axis  is  vertical. 
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6 . 3 _ Synthesized  Perspective  Views. 

The  stick  figure  model  and  the  digitized  image  were  used  to 
generate  a  sequence  of  perspective  views  with  the  computer  program 
described  in  Section  5.  The  outputs  were  sharpened  using  digital 
image  processing  techniques. 

Figures  6-4  thru  6-7  simulate  a  camera  100  feet  across  the 
street  from  the  sunny  face  of  the  building,  with  the  optical  axis 
aimed  at  a  point  on  the  building  10  feet  above  street  level.  In 
Figure  6-4,  the  camera  is  10  feet  above  street  level  and  has  no  tilt. 
In  Figure  6-5,  the  camera  is  70  feet  above  the  street,  nearly  at  the 
height  of  the  roof  and  has  moderate  tilt.  In  Figure  6-6,  the  camera 
is  170  feet  above  the  street,  tilted  significantly  downwards.  Figure 
6-7  simulates  what  a  camera  would  see  if  it  could  be  placed  70  feet 
below  street  level. 

Figures  6-8  thru  6-11  simulate  a  camera  70  feet  diagonally 
opposite  the  corner  of  the  building,  going  through  the  same  sequence 
of  heights  and  tilts  as  the  previous  set. 

Even  though  a  human  may  tilt  his  head  in  viewing  a  tall 
building,  the  untilted  views  in  these  figures  are  more  comfortable  to 
view  than  the  tilted  ones.  Figures  6-12  thru  6-14  illustrate  three 
views  that  an  observer  might  see  as  he  walks  down  the  sidewalk 
opposite  the  shady  side  of  the  street.  A  45°  pan  angle  was  chosen  as 
a  reasonable  approximation  to  the  forward  viewing  of  the  observer. 
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Face-on  View 


Face-on  View,  70  Foot  Altitude,  Moderate  Tilt 


v  ' 

V  I 

•  •  v 

M  *  * 

n  n 

n 

n 

w 

*  *  * 

n 

Ml 

n 

* 

Figure  6-6.  Face-on  View,  170  Foot  Altitude,  Significant  Tilt 
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Figure  6-7.  Face-on  View,  Below  Ground  Level,  Upward  Tilt 
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Figure  6-9. 


Diagonal  View, 


70  Foot  Altitude,  Moderate  Tilt 
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Figure  6-13.  Second  View  of  Sequence 
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8.  GLOSSARY. 

CAMERA  MODEL.  The  pan  (6),  tilt  ($),  roll  (:),  focal  length  (f), 
and  position  (x,y,z)  of  a  camera. 

T 

CROSS  PRODUCT.  The  cross  product  of  two  vectors  (an ,  a0,  a,)  and 

T  1  z  J  T 

(b^,  b2 »  b^)  is  the  vector  (^2b3  -  a3^i  ”  a1^3'  a  iiD2  ~  a2bi^* 

DIGITAL  IMAGE.  An  array  of  numbers  that  represent  the  intensity  of 
an  image  at  discrete  locations  over  a  two-dimensional  grid. 

GRAY  LEVEL.  The  intensity  value  associated  with  a  picture  element 
(pixel) . 


IMAGE  COORDINATES.  The  line  and  sample  coordinate  system  that  can  be 
used  to  identify  a  picture  element. 

IMAGE  MODEL.  The  camera  model  and  scanning  parameters  associated 
with  a  digital  image. 

INNER  PRODUCT.  The  inner  or  dot  product  of  two  vectors  (a.,  b. ,  c.)T 

rp  III 

and  (a2f  b2»  C2)  is  the  real  number  +  ^2.^2  +  clc2' 

MICROMETER  (urn) .  One-millionth  of  a  meter. 

PERSPECTIVE  TRANSFORMATION.  A  geometric  projection  of  a  scene  formed 
by  passing  a  ray  from  each  scene  point  to  a  center  of  projection  at  a 
finite  distance  from  the  scene,  and  finding  the  image  formed  by  the 
intersection  of  the  rays  with  a  plane  of  projection. 

PICTURE  COORDINATES.  A  coordinate  system  for  measuring  points  on  a 
picture.  The  coordinates  are  in  linear  units,  such  as  feet  or  meters, 
as  opposed  to  the  line,  sample  system  of  image  coordinates. 
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8.  --  Continued. 

PIXEL,  OR  PICTURE  ELEMENT.  A  single  element  of  a  digital  image  that 
can  be  identified  by  a  (line,  sample)  coordinate  and  a  gray  level 
value . 

SCANNING  PARAMETERS.  The  scanning  parameters  relate  image  coordinates 
to  picture  coordinates.  They  include  the  linear  unit  measures  of  the 
sample  spacing,  and  the  x-  and  y-  offsets  between  the  pixel  at  (line  1, 
sample  1)  and  the  picture  coordinate  origin. 

SPATIAL  COORDINATES.  A  three-dimensional  coordinate  system  whose 
components  are  measured  in  linear  units. 
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